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(54) Abstract Title: A system for communicating user emotion 

(57) A communications system has two or more nodes, at least one of the nodes comprising an emotion input 
device for user selection of a stored emotion type and a transmitter for transmitting the selected emotion 
type to output devices at other nodes. An output device may be a display for displaying an image 
representing the selected emotion type, e.g. an icon depicting a smile or a grimace. In this way, a 
participant in the communication may indicate a currently felt emotion to the other participants. This will 
be particularly useful in a video conference. The emotion input device may detect user emotions by 
movement of the input device alone, e.g. a mouse, virtual-reality glove, or computer-controlled camera. 
The output device may, alternatively, provide a sensory output such as a sound, music, a smell or a 
physical force such as vibration. 
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At least one drawing originally filed was informal and the print reproduced here is taken from a later filed formal copy. 

Original Printed on Recycled Paper 
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A COMMUNICATIONS SYSTEM 

This invention relates to a communications system. 

It is well known that in face to face communication it is not 
5 merely the speech that is important but the emotions displayed on the 
faces of those taking part and body language. When direct 
communication is not possible, and communication takes place via a 
telephone, the facial expressions are not seen and the communication 
is not as effective. 

10 Video conferencing seeks to overcome the limitations by 

enabling the faces of the participants to be seen. However, because of 
bandwidth restrictions it is not always possible for the facial 
expressions to be clearly seen so in effect video conferencing does not 
always satisfactorily address the problem it seeks to overcome. 

15 Furthermore, not all parties may have the required video conferencing 
equipment or access to a communication line having the required 
characteristics. 

According to the invention there is provided a communications 
system comprising two or more nodes at least one of the nodes 

20 comprising an emotion input device for user input of an emotion type; 
a transmitter for transmitting the input emotion type to the other of the 
nodes and an output device at the other node for outputting a 
representation of the input emotion type to the user. 

The emotion input device may detect user emotions, via the 

25 movement of the device alone. Thus the use of the emotion input 
device is more akin to natural body movements which occur when 
humans communicate. The emotion input device may be any device 
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that can detect and quantify user motion. As such a range of devices 
may be employed as an emotion input device; from the PC mouse, 
through Virtual Reality gloves, to automated visualisation via 
computer controlled cameras. 
5 Alternatively, the input device may enable user selection of an 

emotion type from a number of stored emotion types. 

The output device may be a means to provide a sensory output 
such as a sound, music, smell, a physical force such as a vibration or 
some other means that provides sensory stimulation. In the described 
10 embodiment the device is a display to display an image conveying the 
selected type of emotion. 

The image may be an icon depicting say a smile or a grimace or 
some other image indicative of a particular emotion. 

A specific embodiment of the invention will now be described, 
15 by way of example only, with reference to the accompanying drawing 
in which: 

Figure 1 shows a communications system in accordance with the 
invention; 

Figure 2 shows a communication node of the system shown in 
20 figure 1; 

Figure 3 shows in schematic block diagram form the 
architecture of the communication node shown in figure 2; 

Figure 4 shows blocks of functionality furnished by the 
architecture shown in figure 3; and 
25 Figure 5 shows a display of a node with a video-conference 

taking place. 
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As is shown in figure 1, a communications system 1, in 
accordance with the invention, comprises a number of network nodes 2 
to 5 interconnected via a telecommunications network 6. 

Each network node 2 to 5 comprises a computer terminal 7 

5 having a display 8, a keyboard 9, a mouse 10, a connection 11 via a 
modem to the network 6 and a camera 12 as is shown in figure 2. In 
addition to the usual computer operations the mouse 10 also provides 
an emotion input device in a manner to be described later. (Other input 
devices may be used such as keypads or pressure sensitive devices.) 

10 As will be understood by a person skilled in the art, the 

computer terminal 7 is formed by a computer architecture such as is 
depicted in figure 3. A processor 13 is connected to input devices 14 
such as the mouse and keyboard to memory 15 and output devices 16 
such as the display. The memory 15 provides storage space for the 

15 program controlling the operation of the processor 13. In this 
particular case an application is held which enables video-conferencing 
of the terminals. 

The application provides, in conjunction with the processor 13, 
the blocks of functionality shown in figure 4. The core function is a 

20 video conference processor 17 which enables the connections to the 
participants and monitors for inputs from the keyboard 9 and mouse 
10. The camera 12 provides its video output to the video conference 
processor 17. If the image is to be transmitted to the other participants 
it is encoded and packetised before being passed to a transmitter 18 

25 and a modem 1 9 and thus via the telecommunications network 6 to the 
participants. 



4 



The video conference processor 17 receives information from 
the other nodes involved in the conference call via a receiver section 
20 which is also connected to the modem 19. It strips the information 
from the received packets and passes part of the received information 
5 concerning participants' input emotions to an expression processor 21. 
The local user may also input emotion types via the mouse which is 
also connected to the expression processor 21. The expression 
processor 21 selects from expression memory 22 an expression graphic 
which is passed to the display driver 23. The display driver is 
10 connected to the display 8. It is also connected to the video conference 
processor 17 to receive decoded video information received from the 
camera 12 and from the other nodes involved in the conference. 

A typical conference display is shown in figure 5 and it 
comprises a first region 24 and a second region 25. The first region 24 
15 displays the video information received from a camera at one of the 
nodes. The region 25 displays images representing emotions input by 
the other participants. For participant A, the icon displayed indicates 
an input emotion of agreement. For participant B, the icon displayed is 
one indicating a confused state. For participant C, we have an icon 
20 representing disagreement and participant D, the icon represents 
boredom. 

The images in region 25 are not fixed but are updated as the 
input made by the participants changes. 

In the described embodiment, a number of actions may be input 
25 by the movement of a cursor on the display by the mouse and 
activation of the right or left button or a middle scrolling wheel. The 
actions appear in the tables below. 



5 



In the first table 1 the action is a reciprocating line in which the 
cursor is moved from left to right and then back right to left. That 
basic motion can be varied in its characteristics as shown under the 
appropriate descriptions. Table 2 shows other actions. Each action 
5 may be attributed to a particular emotion type. 
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Table 1 



Action 



Reciprocating 
line 



Example I Characteristic 



Left 



Right 



Direction 



Up 



Down 



Short 



Long 



Description 



Lateral line, started 
from right side 



Lateral line, 
from left side 



started 



Transverse line, 
starting from bottom 



Transverse line, 
starting from top 



Length 



Shortening 



Growing 
Very fast 



Mean length of line 

over last cycle 

Mean length 
decreasing over 
cycles 



Mean length 
increasing over cycles 



Cycle duration fast 



slow 



Up 



Displacement 



Down 



Start end 



Turn end 



End emphasis 



Out leg 



In leg 



Repetition 



>4 



Time to return to start 
another cycle 



Mean line middle 
moves up over cycles 



Mean line middle 
moves down over 
cycles 



Moves slower at start 
end of the line 



Moves faster at start 
end of the line ' 



Moves slower out 
from start end of the 
line 



Moves faster out 
from start end of the 
line 



Number of cycles 
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Table 2 



Action 


Example 


Characteristic 




Description 


Unidirectional 
line 






Left 


Lateral line, started 
from right side 


Right 


Lateral line, started 
from left side 


Up 


Transverse line, starting 
from bottom 


Down 


Transverse line, starting 
from top 


Length 


Short 
Long 


Length of line 


Duration 


Very 

-foot 

rasi 
fast 
slow 


Time to complete line 


End emphasis 


Start 
end 


Moves slower at start 
end of the line 


Turn 
end 


Moves faster at start end 
of the line 


Repetition 


1 

2-4 
>4 


Number of times 
repeated 
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Action 


Example 


Characteristic 




Description 


Single 

enclosed 

space 






Left 


Anticlockwise motion 


uirecuon 


Right 


Clockwise motion 


Q 




Small 
Large 


Mean diameter over last 
cycle | 


Diameter 


Shrinking 


Mean diameter 
decreasing over cycles 




Growing 


Mean diameter increasing 
over cycles 




Cycle duration 


Very fast 

fast 

slow 


lime 10 return iu »uui 

dilUUlCl wjr^l^ 


ui spiacemeni 


Up 


Mean centre moves up 
over cycles 




Mean centre moves down 
over cycles 


End emphasis 


Start end 


Moves slower at the top 


Turn end 


Moves faster at the top 


Repetition 


1 

2-4 
>4 

1 


Number of cycles 


Action 


Example 


Characteristic 




Description j 


Double 

enclosed 

space 


OQ 


Direction 


Lett 


Lateral layout, started 
from right side 


Klgnt 


Lateral layout, started 
from left side 


Up 


Transverse layout, 
starting from bottom 


LHJWII 


Transverse layout, 
starting from top 


Length 


Short 
Long 


Mean length of shape 
over last cycle 


Shortening 


Mean lengtn decreasing 
over cycles 


Growing 


Mean length increasing 
over cycles 


Cycle duration 


Very fast 

fast 

slow 


- Time to return to start 

- another cycle 
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Displacement 


Up 


Mean middle moves up 
over cycles 


Down 


Mean middle moves 
down over cycles 


End emphasis 


Start end 


Moves slower over the 
start part of the shape 


Turn end 


xAnvpQ faster at start Dart 
of the shape 


Out leg 


Movm slower out from 
start end of the shape 


In leg 


Moves faster out from 
start end of the shape 


Loop bias 


Start 


Loop at start end is 
larger 


End 


Loop at start end is 
smaller 


Repetition 


1 

2-4 
>4 


Number of cycles 



It may be that the device provides additional means by which the 
user may express their emotions. Typically these might include; left 
button, right button, scroll wheel. 



These can be used in isolation, in combination, or together with a 
motion action, to express a specific emotion. Some examples of device 
actions templates that could be used are given in table 3. 
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Action 

Wheel 
toggle 

Wheel 
flick 

Button 
click 



Button 
hold 



Characteristic 

Slow 



Fast 
Down 



Up_ 



Repe tition^ 
Left 



Right 



Repe tition^ 
Left 



Right 



Duration 



Repetition 



Description -— , 

The scroll wheel is moved backwards and forwards 

about a quarter turn 
The scroll wheel is moved about half a turn in one 
direction 

A button is pressed and released in less than half a 
second 



A button is pressed and held for longer than half a 
second 



By way of illustration table 4 indicates some possible emotion 
types and how they could be associated with the use of the emotion 
input device. 
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Table 4 



ILIIIOUUII 


Screen pointer device action 


INOrillal 


Pointer stationary (or not recognised as a defined 
motion) 


A ctrf*f* 

AVgrcc 


Slow transverse reciprocating line 


ICa! 


Fast transverse reciprocating line 


TV* o o nrrAP 

UlSaglCc 


Slow lateral reciprocating line 


INL/! 


Fa<;t lateral reciorocatinE line 


uncenain 


Small slow lateral double enclosed space 




Lateral double enclosed space with start loop bias 


questioning 


T arap fa«rt lateral double enclosed space 


IN ecu more lnionnauon 


Tranwpr^e double enclosed space 


Distracted 


Pnintpr focus moves to another application 


Impatient 


Qlrwu whppl topple 


Frustrated 


Poet whppl topple 


taking note 


VpvhoarH pntrv with focus on the application 


Attentive 


TVi»n cvprQp unidirectional line 


Interruption 


Fast reoeated transverse unidirectional line 


Attention request 


Right button click 


Attention urgent 


Repeated right button click 


Amazed 


Right button hold 


Discount 


Wheel flick 


Happy 


Large slow single bounded space 


Sad 


Slow lateral unidirectional line 


Angry 


I pft and right button click 


Aggressive 


Repeated left and right button click 



Each terminal may be trained by a particular user to attribute 
particular actions to emotions. In the training, an emotion type will be 
5 presented to the user and the user will manipulate the mouse in a way 
that the user wishes. Default options are provided in the event that the 
user does not wish to undergo the training operation. 

The images to be displayed to depict a particular emotion may be 
tailored by the user by selection from an image set or in other ways. 
10 Different image sets may be provided for different cultures. For 



12 



example, a western set, an oriental set or a middle eastern set may be 
provided to cater for the cultural heritage of the participants and to 
avoid offence being caused. 

Once the image is selected the expression processor 21 informs 
5 the video conference processor 17 which transmits the emotion image 
identification to the other participants in the conference in an 
information packet. The packet includes the identity of the user 
expressing the emotion, a timestamp with the local time and the time 
since the application was first linked to the conversation, the emotion 
10 image identifier, suggested image set, the duration of the emotion. In 
some applications a URL of a custom image may be transmitted to 
select a previously used image. 

Whilst in the described embodiment, a video conference is 
established, the invention is applicable to other situations. For 
15 example, a conference call may be established and only the region 25 
of the display is used to show the images representing the input 
emotion types. The first region will then be blank. This then caters for 
the situation where video cameras are not available. 

In a similar manner, a simpler system is envisaged in which a 
20 telephone is associated with a PC which provides a mouse that is used 
to express a particular emotion type. The emotion type may then be 
displayed to similarly equipped participants via the PC display. 

Another example is where the emotion type expressed is 
associated with the user's "presence" data that may be accessed by 
25 other users both prior to and within a communication. 
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In the described embodiment the emotions are displayed as icons. It 
may be possible to use digital images of the participants in alternative 
embodiments. The images will comprise a set of photographs of the 
participant in different applicable emotional states. These images may 
be transferred when communication is established. 
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CLAIMS 

1 . A communications system comprising two or more nodes where 
5 users at the nodes may communicate with users at the other nodes, at 
least one of the nodes comprising an emotion input device for user 
input of an emotion type; a transmitter for transmitting the input 
emotion type to the other of the nodes and an output device at the other 
nodes for outputting a representation of the emotion type to the user. 
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2. A system as claimed in claim 1 wherein the output device 
outputs a representation in the form of at least one of: an image, a 
smell, a force or other user perceivable output. 

15 3. A system as claimed in claim 2 wherein the output device is a 
display which displays an image as a representation of the selected 
emotion type. 

4. A system as claimed in claim 1 wherein the emotion input device 
20 enables a user to make a selection of emotion based on their body 

movements. 

5. A system as claimed in claim 4 wherein the emotion input device 
comprises a computer mouse. 

25 

6. A system as claimed in any preceding claim wherein the 
transmitter transmits an identifier of the selected emotion. 



• * 15 

7. A system as claimed in claim 6 wherein the transmitter transmits 
an identifier of an image set from which emotions are to be selected. 

8. A communications system as hereinbefore described with 
5 reference to, and as illustrated by, the accompanying drawing. 
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